The Hard Thresholding Pursuit (HTP) is a class of truncated gradient descent methods for finding sparse solutions of $\ell_0$-constrained loss minimization problems. The HTP-style methods have been shown to have strong approximation guarantee and impressive numerical performance in high dimensional statistical learning applications. However, the current theoretical treatment of these methods has traditionally been restricted to the analysis of parameter estimation consistency. It remains an open problem to analyze the support recovery performance (a.k.a., sparsistency) of this type of methods for recovering the global minimizer of the original NP-hard problem. In this paper, we bridge this gap by showing, for the first time, that exact recovery of the global sparse minimizer is possible for HTP-style methods under restricted strong condition number bounding conditions. We further show that HTP-style methods are able to recover the support of certain relaxed sparse solutions without assuming bounded restricted strong condition number. Numerical results on simulated data confirms our theoretical predictions.

This paper is quite technical but is very interesting and promising as it provides sufficient conditions for exact recovery or sparsistency of the solutions of an quite broad class sparsity constrained smooth optimization problems using a very simple, yet powerful, class of greedy algorithms. Despite the technicality of the results, they are very clearly synthesized and nicely presented. I have only a few (but important) concerns: Line 26: Problem (1) with a squared regression loss is IMO more general than compressed sensing. Depending on the forward model, it can tackle any noisy linear inverse problem such as optimal sparse representation in a dictionary, denoising, deblurring (or deconvolution),... Line 32: There is also that paper: [1]A. Line 81: least squared - least squares Line 121: In the definition of x {-*} the constraint with strict inequality leads to something that is ill-posed.

The Hard Thresholding Pursuit (HTP) is a class of truncated gradient descent methods for finding sparse solutions of $\ell_0$-constrained loss minimization problems. The HTP-style methods have been shown to have strong approximation guarantee and impressive numerical performance in high dimensional statistical learning applications. However, the current theoretical treatment of these methods has traditionally been restricted to the analysis of parameter estimation consistency. It remains an open problem to analyze the support recovery performance (a.k.a., sparsistency) of this type of methods for recovering the global minimizer of the original NP-hard problem. In this paper, we bridge this gap by showing, for the first time, that exact recovery of the global sparse minimizer is possible for HTP-style methods under restricted strong condition number bounding conditions. We further show that HTP-style methods are able to recover the support of certain relaxed sparse solutions without assuming bounded restricted strong condition number.

In the past decade, high-dimensional data analysis has received broad research interests in data mining and scientific discovery, with many significant results obtained in theory, algorithm and applications. The major driven force is the rapid development of data collection technologies in many applications domains such as social networks, natural language processing, bioinformatics and computer vision. In these applications it is not unusual that data samples are represented with millions or even billions of features using which an underlying statistical learning model must be fit. In many circumstances, however, the number of collected samples is substantially smaller than the dimensionality of the feature, implying that consistent estimators cannot be hoped for unless additional assumptions are imposed on the model. One of the widely acknowledged prior assumptions is that the data exhibit low-dimensional structure, which can often be captured by imposing sparsity constraint on the model parameter space. It is thus crucial to develop robust and efficient computational procedures for solving, even just approximately, these optimization problems with sparsity constraint.